DOCOHENX RESOHS 



ED 201 337 



IB .009 351 



AUTHOR 
TITLE 

INSTITOTION 

SPONS AGENC7 

BEPORT NO 

POB.DATE 

GRANT 

NOTE 

AVAILABLE FROM 



EDRS PRICE 
DESCBIPTOES 



Hay€s-Rothr Frederick; And Others 
'Knowledge Acquisition, Knowledge Programming, arid 
Knowledge fiefineoent. 
Band Corp,, Sant& Honica, Calif « 
National Science Foundation, Rashington^ D«C« 
ISBN- 0-8330- 01 95-7; Rand-R-25ttO-NSF 
Hay 80 
MCS77-03273 
39p, 

Rand Corporation, Hain St., Santa Clonica, CA 90^06 
($3.00) . 

MF01 Plus Postage.. PC Not Available from EDRS. 
♦Artificial Intelligence; *Computers; Inf occiatiDi 
Theory; Man Machine Systems; ^Programing; Research 
Reports; Systems Development 



ABSTBACT 

This report describes the principal findings and 
recommendations of a 2-year Band research project on machiae-aidsd 
knowledge acquisition an^\Mscusses the transfer of expei;tise from 
humans to machines, as- well^Xsthe functions of planning, debugging, 
knowledge refinement, and autonSmotis machine learning. The relative 
advantages. of humans and machines in the building of intelligent 
systems f.re explained. Background and guidance is provided for 
policymakers concerned with the research and development of 
machine^ based learning systems; The research method adopted 
emphasized iterative refihement of knowledge in response t^ actiial 
experience; i«e., a machine's knowledge was acquired initially from a 
human who provided enough concepts, constraints, and problem-solving 
heuristics to define some minimal level of performance. Sixty-two 
references are listed. (Author/FM) 
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PREFACE 



This report describes tb r ncipal findings and rtscocnmendation:: of a two-year 
Rand reaeai-ch project on r .nine-aiderf iDOwJedge acquisition, supported by a 
grant fr-^ tae Intelligent S^^iems Progranw Mathemuitical and Compner Sciences 
Division National Science^ bundaiion. Itdscu^s ths: transfer of ez^ertise from 
humans cc :nachines, as w^_ l as the functfonn c; planrring, debugginz. knowledge 
refinement, and autoriomoi5.i machine leamim Thrri^iout, the aunrDrs attempt 
to explain the relative aiysTiirsgeE of hiiimatai naachines r i tfe building of 
intelligent systems. This Vx- ^^'^oirn mcr^^'^t^ acariproach to nvbrid systems in 
which humans provide initi?-^ i:r.£::?Eedg'el3 nnrcihint^s and cnopersse wStli them in 
the iterative evaluation and Teiinsnem-;af tteiiE30»>rjedge4aHs sysism. The report 
provides background and ,uidanc:s for jx^-.'^ . makers concerned with tte research 
and development of mach:ne-based lea. .nng. fr>'sterts, and it should be of interest 
to computer scientists, cognitive "Dsychruasisv znd others interested tn machine 
intelligence. 

During this project, the research strr r zziuored nruix : alternative? s^Ti^proaches 
to machine-aided learning and cor^r-uuHi z:rszt]iie uost prnr".isir :^ aT^nv^t^^'or future 
work is to translate hunian experns-^ nsrr niizial irstelligent: program ^^ad then to 
refine the implementabons itefaLiv?iv \2B:iKScatet by «JX^perience, !7iu.d:ipproach 
blends machine capabilities with humim^xpertiae ir. th^ generatior and iterative 
improvement of intelligent systen. 

The report explains why the authnrr have incssenr : explore ihis problem, 
describes how they are pursuing tneir Htmk, and iscrujsses the pvci:^\xi s to date. 
Because the stated task is very diSciik, rife isurfwo "las onl:^ been scn>i:taiied; much 
remains to be done. 

Severial computer programs nave bsi ievelcJO^ in the course if this project 
by various combinations of the authors ^eir c illabocators. Mossssv has imple- 
mented the. knowledge-programming s^jem ?: rt- of his dissension research 
supervised by Hayes-Roth. Hayes-Rotti and KEahv liave iinplemeiitsd three difer- 
ent programs for representing and apprrx^^ the:.:na^ie<?xg?of the .ifiatrts don^aia. 
While these programs actually play the i::^^^ nriodtjrately well, their 'nriBrest for as 
lies in their altemativ3 approaches to ki:rtwJeds^rep*^^#ertation and control. Lenat 
has implemented in his program RLL som#i ^^^the idt^r. for restructuring knowl- 
edge discussed in.Lenat, Hayes-Roth, and K ihr ISTSr & 1979b). All of these 
programs should be considered fundameniiall:^ ai tsKpe^t^entsi and ephemeral; 
•they serve primarily as 'iDreadboards" for testing oiz' ideas. In the future, they may 
give rise to additional experiments or more ^^msn^nt and extensive applied sys- 
tems. ^ 

David J. ^ostow is a doctoral candidate ir ^ze j /epartment of Computer 
Science, Carnegie-Mellon University, and a consultant t The Rand Corporation. 




SUMMARY 



The scientist who wants to undcjrstand and construct learning systems must 
first define "learning'' and then disvelop a research method that eventually pro- 
ducesieaming mechanisms. In contrast to conventional vie^s of learning that focus 
oti problems of feature selection, combination, anci weighting, the definition we 
have used in this study emphasizes a continual grovvth and refinement of knowl- 
edge. From this point of view, a system learns By increasing the scope, precision, 
validity, and power of its current knowledge. That knowledge includes descriptions 
and models of the environment, is well as planning and problem-solving techniques 
for achieving goals. Learning requires the system to augment, generalize, special- 
ize, validate, and ex ploit the knowledge it possesses at any point in time. 

The research method adcjpted here emphasizes iterSi^^l^finemenfof Tcnow 
edge in resiponse to actual eiperience. In this approach, a machine's knowledge is 
acquired initially from a human, who provides enough concepts, constraints, and 
problem-solving heuristics to define some minimal level of performance. We use 
semia?itomatic methods to convert the initial knowledge into a working program. 
Then^ when the initial progi-am executes, we observe the resiilting behaviors to 
diagnose problems and design knowledge refinements. Although this refinement 
process currently requires significant hiunan contributions, we likve formulated 
methods that may make it possible to reduce or eliminate much of that involve- 
ment. 

This report explains the cyclic roles of understanding, planning, and evaluation 
during the development and extension of knowledge. A prerequisite for learning 
:s an initial base of knowledge. Most learning results'* from the constructive appli- 
cation, evaluation, and refinement x)f that knowledge. By predicting probable conse- 
quences of our actions and noticing unfulfilled expectations, we can isolate and. 
often quickly remedy weaknesses in existing knowledge^ In this framework, a 
learning system, must use existing knoWledp^e to plan reasonable courses of action, 
carry those plans out, and then diagnoss ^'ye/:/ 'T?ess€s that explain observed failures 
or unexpected successes. The processes o*'|XnUva'r5g, acting, and evaluation cooper- 
ate to produce new knowledge by refinin > " ^- rl extending prior knowledge. 

The suggested approach is illustratf^d b > he following simple example: Let us 
suppose that an intelligent system has been aesighed to control a power generator. 
Its knowledge base describes the system's likely behaviors, sensor readings, and 
control mechanisms. These behavioral specifications then relate assumed situation 
conditions to optional control actions and their expected effects. For example, the 
knowledge base might describe, the relationships among several valves, pumps, 
pipes, temperature gauges, and control switches. Let us assume that rising temper- 
matures produce some undesirable situation D. A plan to compensate for rising 
temperatures indicated by sensor A might dictate turning off valve B on*pipe C. 
This plan would be associated with a rationalization, i.e., a proof or informal argu- 
ment, establishing that when A indicates rising temperature, closing B on C will 
counteract the^undesirable consequence D. The proof might also establisli addition- 
al expectations ^bout corroborativ^^i sensor readings or side-effects produced by the 
B closure. Let us\iext sil^pose that, subsequently, /D actually occurs. We can now 
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compare the assumptions and deductions in the rationalization with actual obser- 
vations of the current situation. We may find that all the assumptions remain true, 
but some deductions are falsified by data. In this case, we can identify the faulty 
inference rules as those whose premises are true but whose conclusions are false. 
Alternatively, we may find that some of the expected or presupposed conditions do 
not obtain. In this case, too, we can identify the source of iie faulty expectation. 
Once it has been identified, the fauny rule becomes the su:::nect of refinement. 

In refining a faulty inferential rule, a variety of heurisrics may be used. Each 
potential change to existing knowledge reflects a new hypothesis about thebehav- 
ior of the modeled system or the effects oi' actions upon that system. These hypotne- 
ses constitute the intelligent system's new knowledge. To illustrate any of these 
learning heuristics requires an actual comparison between expectations and obser- 
vations. But let UB suppose that the plan for preventing D reasoned as follows: 
Closing B prevents water from reaching E (B-E rule), which, in turn, prevents vapor 
from saturating F (E-F rule);' and only a saturated F could cause D to occur (F-D 
rule) -Now a variety of diagnoses and refinements might arise. To pursue one of 
these, suppose that F does~not saturate but D occurs; we should reject the current 
F-D rule. To refine this rule we would need to specify more accurately those situa- 
tions in which the predicted F-D relationships occur. We might then use any of 
several available techniques for constructing descriptions capable of distinguishing 
the current situation from earlier ones in which the F-D relationship behaved as 
predicted. The new discriminating specification would yield a new hypothetical rule 
such as,. When condition X occurs, only saturating F causes D. 

Alternatively, we may find a source of improved knowledge In the proof that 
underlies the plan. This proof might introduce some unproved and invalid assump- 
tions. In the current example, the belief in the eflfectiveness of closing B for preclud- 
ing D mijght rest on the assumption that saturating F causes it to short-circuit and, 
implicitly, that nothing else could. In practice, many assumptions of this sort enter 
into planning quite implicitly. But in this case, a short-circuited F unit in the 
presence of nonsaturation reveals the faulty assumption directly. Many more illus- 
trations and examples of the diagnosis and refinemen*: process are described in this 
report. 

Some of the ideas described here have been implemented in worjking computer 
programs that are described in detail elsewhere; however, in this study we have 
concentrated primarily on improved problem formulations, concept developments, 
and ''hand-simulations." We anticipate that long-term research will be required to 
implement working computer programs that can perform all the functions this 
research paradigm, suggests. 
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We grateful ^' ickr.aw:edge ui ranLiTe contributions cf our Rand, Stan- 
ford, and Camer Mtion :ciiea|i::ir5 t:is work. John Burg(=j collaborated with 
us on learning hHirristica. SnzaBej r -'J^n^^jnieia made frequent contnoutions to our 
efforts on knowii^sse repreficoissT^o. i^^ - -^ksLenat assisted in the cEvelopment of 
knowledge reprs?^*^taxions, /tiwuns::^ rfd ^ocedures for cognitiv^f economy. As 
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understanding c i^^ationaiir .acn: lii e sport, however, represen ;3 only the au- 
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1. BACKGROUND: MACHINE-AIDED KNOWLEDGE 
ACQUISITION IN PERSPECTIVE 



The processes of knowledge acquisition and learning have fascinated peopi^ 
throughout much of history. In this century, the growth of behavioral and informa- 
tion sciences has stimulated various forms of basic and applied learning research-. 
Behaviora) psychology, for esample, has made impressive gains in developinr 
practical procedures ifor improved training of humans anB animals. This zype . 
work focuses primarily on the nature and the appropriate timing of contingsn: 
reinforcements. The essential finding has been that a reward received soon aft^rr 
some desirable behavior occurs increases the chance that the behavior will recur. 
Cognitive psychology, on the other hand, emphasizes the ideas and concepts gDv- 
erning ordinary thought. For example, researchers ir^ this field attempt to explain 
hov/ humans induce common category concepts such as "dog" or "criminal" frcm 
examples. Researchers in the relatively new field of artificial intelligence (AI) !::£ve 
addressed both of these kinds of learning problems. However, their emphas ^ on 
machine learning imposes demanding constraints on potential theories, Fc AI 
purposes, a theory of learning must lead to a computer program that exh bits 
improved performance over time. 

This report describes only one of many approaches to machine learning. ^Thus, 
we shall first review briefly the related AI research that forms the current scie^itific 
context. The several approaches that have developed over time have emphasized, 
in order, adaptively a^justiijg feature weights; generalizing examples of categories, 
transformations, and more general procedures; using heuristics to synthesize new 
concepts; and directly transferring human knowledge to computers. 

Early AI work studied, adaptive learning schemes that dould adjust control 
parameters to correlate the machine's output with a desired standard. In this sense, 
the early learning devices acted somewhat likQ adaptive control devices. The Per- 
ception (Minsky & Papert, 19G9), for example, was a pattern-recognition device 
that classified test patterns by computing weighted sums of feature-detector 
outputs. When the sum exceeded some threshold, the response indicated corre- 
sponding class membership; if the sum fell below the threshold, the pattern was 
rejected frolih the class. When an incorrect decision occurred, a learning j^lgorithm 
prescribed how to adjust the feature weights. Because the machine a'djwsted its 
weighting factors to accommodate its training experiences, this kind of learning 
might be considered the first of many subsequent paradigms for "learning by 
example." (A different appHcation of a similar technique addressed tactics in 
the game of checkers (Samuel, 1963).) 

Over time; AI researchers moved increasingly toward a belief that intelligent 
behavior requires substantial world knowledge. Most intelligent tasks require spe- 
cific feature^etectors, complex descriptions of patterns and structures, and corre- 
spondingly complicated procedures for comparing one description with another. 
Without these, very few h^man capabilities could be simulated. Soon after the 
development of behaviort^l learning theories and devices Hke the Perceptron, scien- 
tists began to point out the need for these more complex mechanisms. Work in 
machine-visiop programs, for example, established the need for specialized detec- 
^tors for edges, comers, and intersections and sophisticated procedures for fcllovnng 
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and interpreting their connecr:nr-_i_ scenes. These kinds of ir :emal structures lay 
outside the scope of the earlic f:.;rr_ing frameworks. A number of AI researcher^ 
then developed improved iT)e: ::jr learning by example ^hat could generalize 
rules from arbitrarily comple:: ^iirrmiral descriptions (Buchanan et al, 196S; Bu- 
chanan & Mitchell, 1978; Havri^^iri::, 1974; Hayes-Roth, 1576a; Hayes-Roth, 1976h; 
Hayes-Roth, 1977; Hayes-Rozi iT^; Hayes-Roth & Burge, 1976; Hayes-Roth & 
McDermott, 1976; Hayes-Rot ik iVi.-Dermott, 1978; Mitchell, 1977; Soloway & Rise- 
man, 1977; Vere, 1978a; Vere. ISTSb; Winston, 1975). These procedures use initially 
provided feature detectors ar weL as "structural" or "relational" connections to 
describe each example of a grren class. Then by partial-matching the descriptions 
of many examples, common subdescriptions emerge as candidate general rules 
(Hayes-Roth, ir''8a). This methodolo^' has supported machine induction of trans- 
formational grammar rules (Hayes-Roth & McDermott, 1978), chemical reaction 
rules (Buchanan et al, 1969; Buchanan & Mitchell, 1978), and simple robot plans 
(Vere, 1978a)» among others. 

The basic limitation of trds more recent work derives from its "subtractive" 
approach to learning. The learning programs devised under this approach produce 
new rules by detecting whid: of the currently known features and relations appear 
jointly in each example. By assumption, each example reflects feriterial features as 
well as some irrelevant features peculiar to the specific example. Learning by 

" example, in this context, simply requires subtracting the irrelevant features in each 
case. While this approach can be very useful for i)ractical problems in pattern 
recognition and data interpretation, it provides little insight into the discovery of 
new features or'new functions for performing a task. ■ ' \ _ 

Two recent research projects have shed some light on such discovery problems. 
Exemplary-programming research at Rand (WateVman, 1978b; Faught et aJ,, 1980)^ 
has investigated the problem of inferring programs capable of recreating the in- 
teractions between a human and a machine engaged in a task (see also Biermann 
& Krishnaswamy, 1976). The creation of a program from a human/machine dia- 
logue requires methods that are mdre constructive or '^synthetic" than other learn- 
ing-by-example tasks. The two chief problems concern interpretation of the exam- 
ple' behaviors and the subsequent regeneration of corresponding behavior in new 
contexts. Interpreting an arbitrary human/machine interaction appears to require 

^a variety of sources of knowledge, including sources for explaining (1) the meaning 
of special typed symbols, (2) the state of various systems employed during the 
session, (3) the semantics of computer system outputs, (4) the goal of the person 
performing the task, and (5) the problem-solving procedure that person apparently 
applied. Each of these types of knowledge contributes to understanding both why 
and how the person and the machine cooperated to solve the task. To construct a 
generalized program that can replace the person in such tasks, we must convert 
this passive understanding of the task's purpose and solution methods into effective 
procedures. These procedures, if truly general, must accomplish the same'effects 
although various situational characteristics will differ from the initially observed 
data. This requires several types of knowledge in addition to that previously noted: 
knowledge concerning system control and interactions; knowledge of planning and 
problem-solving; and knowledge of programming methods. 

Another recent project that illuminates the synthetic nature of leariling at- 
tempted to simulate the discovery process in mathematics (Lenat, 1976; Lenat, 



1977a; Lenat, 1977b; Lenat & Harris,-1^78). This project employed -twp .tyt)es' of " 
knowledge to induce n^w concepts of elementary set theory. 'The .first type of 

. ^ - krio\Vledge consisted of a variety of mathematical concepts,- sdch as* sets, lists, 

equalities, arid functions. Over time, the program'3 conceptual kno\yledge gi'eWas 
new concepts were created from existing ones. The methods for discovering new 
concepts constituted the second type of knowledge. A few hundred rules called 
"discovery heuristics" modified existing concepts to produce new ones. For exam- 
ple, several heuristics formulated new concepts by "generalizing'^ ofd. ones. Al- 
though^the program knew at the outset the concepts of a list (defined as "an ordered 
collection of elements") and length, it conjectured for itself a new concept that 
generalized these notions to produce the concept of "length of a list of identical 
elements." In this way, it produced the concept of unary numbers, that is, a list of 
n tick marks meaning the number n. Other rules formed new concepts by specializ- 
ing existiiofi^concepts, by searching for examples of newly conjectured concepts, 5r 
by forming new mathematical functions with arbitrary attributes (e.g., by restrict- 
ing binary functions that applied only to cases where the first and second argu- 
ments were equal). In this way, several insightful and interesting developments of 
mathematical history were retraced .in a few hours of computer time. 

The last type of machine learning we must mention might be called "transfer 
of expertise" (Anderson, 1977; Balzer et al., 1977; Davis et al, 1977; Davis, 1977; 
Davis, 1978; H a yes-Roth e t_aL^978; Heidoni,4Ji26;^ 

^-^^"^^ 1979b; Samuel, 1963; Waterman, 1978a; ' 

Waterman et al., 1979; Barr et al.,' 1979). Work in this area has aimed toward 
.constructing intelligent systems according to heuristic techniques prescribed by 
human experts (Feigenbaum, 1977). The major obstacle to implementing intelligent 
programs described in this w^y arises from the need to translate human knowledge 
into computable formalisms. Several research projects have demonstrated the via- 
bility of using English-like, rule-based languages (Aiifierson & Gillogly, 1976; Davis 
• & King, 1976; Shortliffe, 1976; Waterman & Hayes-Roth, 1978; Waterman et al., 
1979) that enable liumans to express their knowledge in rules of the form. If there 
is a drilling sitejA^hose iron content exceeds 12 ppm and whose location is within 
12 miles of an oil field, then the probability of a moderate iron deposit is high. While 
this particular rule is fanciful, a number of expert systems have been creat'^d for 
problems as diverse as infectious blood7disease therapy (Shortliffe, 1976), artificial 
respirator maintenance (Fagan, 1978), internal medicine (Pople, 1975 & 1977), and 
geological prospecting (Duda et al., 1978). The major lines of continuing effort in 
this context aim (1) to develop improved high-level languages for such rule-based 
programming (Waterman & Hayes-Roth, 1978; Waterpian et al., 1979) and (2) to 
assist in the construction and maintenance of large sets of rules by developing 
metaknowledge — knowledge about the likely and appropriate.kinds of knowledge 
: that should enter the data base (Davis, 1978; Davis, 1979; Stefik, 1979). ' ] , 
The principal concepts of previous learning research that we have discussed are 
summarized in Table 1. The table also gives rough definitions of these terms as well 
as brief descriptions of the mechanisms Al res<^archers have proposed for accom- 
plishing various types of learning. 

Our current research attempts to integrate and extend the best aspects of these 
alternative approaches to machine learning and knowledge acquisition. Specifical- 
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■ Table 1 

Previous Learning Concepts and Mechanisms 



Concept 



Traditional Meaning 



Mephanism 



Learning - 

Adaptive learning 
procedures 

Learning by example 



Ej^eniplary 
programming 



Concept discovery 



Transfer of expertise 



Knowledge 
acquisition 



Changing behavior to unprove- 
performance • 

Changtnc^ behavior output 
to more closely approximate 
desired standard. - y 

4- 

Inf^rring a general classifica- 
tion rule from training sets. 

Inferring a general procedure 
from sample human/machine 
dialogue. 



Inferring the ^istence and 
definition of a general class. 



Supplying human problem- 
solving knowledge to a 
machine. 



Incremental addition of 
knowledge to an intelligent 
system. 



Rote memorization and cone jonhig 
to contingent reinforcement. . \ 

' Adjust association weights ^nnecting 
stimulus fei^tures to outputs by feS'^- , 
back (positiye/negative). ' x '7 

* Propose any boole^ comljiiiation -of ' 
feSktures consistent With the examples 
as the classifying rule. ^ ^ 

Infer the human's purpQSc- that 
motivates the dialogue, interpiSa^ the - 
dependencies between maj^i^e a^d , 
human inputs, then create a program 
to mimic the human. 

Detect that several distinct things :-Oi^' ' 
events share some 'featurQS,'' find a ' 
common description, and^ propose it ^ 
a&i a concept definition;' or Jheuris-^**^ 
tically modify a prior .concept defifti-^ 
tion, conjecture the concepjb^ validity, 
and find examples of It. - 

Program corresponding procedures di- ' 
rectly; or express, the knowledge 
within a narrow formalism sujt^ble 
for machine interpretation; or/expre&^ 
the knowledge in a high4evel t^roblem- 
solvmg language (e.g., a rule-based *• 
system). 

Formulate a representatioi^ for a 
type of Iciiowledge the system use^, 
use any mechanism to^-^dervJiify new 
units of knowledge, and ad^L^Ji^ ta 
the knowledge base. 



\ ( 




ly, we believe the following coniributory factors to learning can and should be 
accommodated in a single system; ^ . \ 

, 1. Contingent reinforcement — when behavior produces undesirable conse- 
quences, the knowledgiB responsible should be altered, and tendencies that 
produce desirable outcomes should be strengthened. " , ----^^ 

2. ^' Learning by example— systems should benefit from and generalize their 

experiences.* . , i : - 

3. Knowledge as the source of power^a learning system should acquire, 
rh^i^ulate, and apply knowledge in the pursuit of increased capabilities. 
The understanding of goals and the capability for planning to achieve 
them. > 

^Acceptance of human advice and knowledge about the: task. 



4. 
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Integrating these five capabilities into a system will require significant effort, 
and only a small proportion of the needed programs have been implemented to- 
date. On the other hand, y/e now understand many of the residual problems clearly. 
Section II explains our formulation of and approach to the problem of understand- 
ing and assimilating knowledge about an arbitrary task. The concepts and mecha- 
nisms that , characterize our approach are summarized in Table 2. Knowledge is 
acquired from humans, used to develop plans for achieving goals, and finally con- 
verted into executable programs. Section III discusses an iterative cycle of plan- 
ning, acting, and evaluation that relates initial knowledge to specific behaviors and 
finally to new or mod^'fied concepts. After relating observed effects to specific prob- 
lematic components Ox ^la^ " and, in turn, attributing these to erroneous 6lements^ 
of domain knowledge, tne system diagnoses deficiencies and conjectures new 
knowle4ge elements. Such knowledge refine;ftents engender a new cycle of plans, 
acts, observed effects, and inductions. Xjeaming, in this paradigm, is equivalent to 
the iterative improvement of performance a];ising from discoveries made whil^ 
implementipg and refining knowledge. > 



Table 2. 

, Lj^arning Concepts and Mechanisms in the Current Approach 



Concept 



Meaning 



^ Mechanism 



Hum^n/knaciiine 
coop^rati'' 



Advi' 



Gcil'dirocted 
planning 



Contingent 
reinforcement) 



Learning By 
^example. 



Knowledge 
refinement 



Human and m achine co- 
operate to build int<?lligent , 
systems and improve them 
ovtr time. ^ 

Humans define domain 
concepts, specify behavioral 
constraints, and suggest 
pvoblijm-solving methods. 

Development of a plan for 
achieving the expressed 
'goals that uses suggested ' 
methods and satisfies ex- 
pressed constraints.^ 



Strengthening of rewarded 
behavioral tendencies 
" and^wakening of others. 

Use of actual situation's^ 
and results to trigger 
learning. 



Amendment or extension of 
knowledge in response to 
behavioral feedback. 



Human- provides initial advice on task 
kKowledge and 'problem -solving meth- 
ods; human aids in converting the 
advice into working progifams and in 
diagnosing and refming knowledge. 

Develop a knowledge' representation, 
. express the knowledge in this form, 
and, integrage it into a working pro- 
gram semiautomatically. 

Work' bapkward from goals to suf- 
ficient conditions and actions by 
deductively pursuing logical, heur- 
istic, and instrumental transformations. 
These transformations symbolically 
manipulate the domain concepts, con- 
straints, and problem-solving methods 
to produce aij effective procedure. 
The planning fj^rocess establishes as- 
sumptions an ^ ^Expectations about 
plan-related situatibns and effects. 

Reinforce knowledge contributing to 
favorable outcomes; diagnose ?nd re- 
fine knowle^dge causing failures. 

Compare expected outcomes. to actual 
outcomes and assumed conditions to 
observed conditions in order to diag- 
nose fallacious planning knowledge 
and to suggest knowledge refinehients. 

Adjust the conditions assumed neces- 
sary or 'sufficient for an action to pro- 
duce an effect;" generalize .or further 
specify the expectations' associated 
with an action. / 
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TWO PARADIGMS FOR DEVELOPING EXPERT SYSTEMS \ 

Many of the recent successful applications of AI have shown the power of t 
implementing, more or less directly, the heuristic rules of human experts. In tasks 
requiring only one or two types of inferential procedure, such as interpreting 
symptoms and test results in medical diagnosis (ShortlifFe, 1976), nearly all domain 
knowledge can conform to a few generic forms or representations. Many appli- 
cations of this type have adopted an if4hen rule form^<^ for expressing the causal 
and inferential relationships (Feigenbaum et al., 1971; Waterman et al., 1979). To 
deriverimplications of known facts, the system apphes the rules to any data that 
satisfy the rule antecedents (the if component), and the corresponding rule conse- 
quents become derived facts. The system infers likely causes o^ observed symptoms 
by reasoning deductively from observations to plausible causes. When the situation 
data match rule consequents (the ihcn components), the system hypothesizes that 
the associated antecedent conditions may also be true. In this way, the system 
reasons backward from effects to likely, causes. 

These systems succeed, in part, because they use constrained rule formalisms 
and perform only one or two specialized kinds of inference. These constraints 
enable program designers to provide naturalistic languages in which nonprogram- 
mers can conveniently express their knowledge. Such English-like languages-make 
it easy for experts in various domains to create large and powerful rule sets by 
communicating directly with the computer. Because of their specialization^ these 
systems can also help in checking the consisteiicy and completeness of the human 
rule. sets. 

The simplicity of these systems, of course, means that they lack the capabilities 
needed for solving most types of problems. Although many domains have problenis 
.similar to the medical diagnosis problem, most intelligent systems need to perform 
a wider variety of actions than interpreting symptoms in terms of their underlying 
causes. If we consider such interpretation tasks as a special kind of perceptual 
process, we can easily see that intelligent behavior involves naore than interpreta- 
tion. Usually, we think of perception, planning, adaptive control, and knowledge 
acquisition as essential components of intelligence. We are presently, unable to 
formulate all of these activities in terms of one uniform type of representation that 
requires only a small number of related inference methods. 

Thus, although a human expert may know exactly what intelligent behavior 
requires in some new domain of interest, the current state of the art requires that 
a human programmer design and implement a unique system for most new tasks.. 
Weeks or months after the initial transfer of knowledge from the expert tp the" 
programiner, a program emerges ready to run.' Figure 1 illustrates this typical 
process. 

Of course, when the program finally runs, it, typically produces a> variety of 
unexpected results. At this point, both the expert arid the programmer discover 
that the original knowledge apparently underspecified the program, because a 
variety of situations produce unanticipated effects. Four types of problems explain 
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Fig. 1— Expert system development and testing 



most of the behavioral deficiencies: (1) The expert neglected to express rules to 
cover all of the special cases that arise; (2) the expert's rules did not produce correct 
conclusions, because they made erroneous assumptions; (3) although the program- 
mer's implementation decisions were consistent with ambiguities in the bri^nal 
specifications, they generated tmdesirable behaviors; or (4) the programmer over- 
looked or incorrectly implemented some of the expert Vad vice. 

Observations of undesirable behaviors motivate a variety of discoveries and 
changes: New knowledge arises from efforts to haindle additional special cases; the 
expert niodifies his initial knowledge to correct the errors in it; theSexpert refines 
his initial rules to resolve problematic ambiguities; the programmer' modifies his 
inference methods or associated program code 'sd that the program behaves as it 
should. Unfortunately, all of these changes require programmer intervention, and 
most of them take significant time and effort.' 

Iterative refinements generally cause programs to become progressively more 
obtuse in their knowledge representations and control structures. This, in turn, 
makes it increasingly difficult for the expert to manage or comprehend his knowl-^ 
edge-based system.! Ordinarily, the costs of programming and refinement are oner- 
ous; few programs' ever satisfy their designers, because design goals continually 
evolve as experience reveals additional system shortcomings. 

' This analysis suggests an alternative paradigm for the jprogramming and itera- 
tive refinement of intelligent systems. The principal components of this paradigm 
are shown in Fig., 2. ^ . " ^ 

This paradigm views t^Q pi^ogramming ptoblem primarily as one of translating 
expert advice into an operational pro-am, and the iterative improvement problem 
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as one of diagnosing program behavior to modify those elements of khowledge tliat 
produce undesired behaviors. This proposed scheme emphasizes the problems of 
understanding high-level ad\dce, converting it into effective behavior, and, inevita- 
bly, changing the knowledge and reiterating the cycle. These problems are referred 
to as knowledge acquisition, knowledge progrmnming, and knowledge refinement, 
respectively.^ For some time to come, all of these processes will require some 
human participation. Thus, throughout this report we will describe semiautomatic 
procedures for^performing these functions.^ 

In the remainder of this section, we explain the primary problems and'proposed 
methods for the first two processes, i.e., acquiring the expert's knowledge by under- 
standing advice and converting this advice into executable programs. These are the 
tasks of the knowledge programmer. In the process of knowledge programming, the 
knowledge programmer develops plans and procedures used by the resulting pro- 
gram. These plans create expectations concerning the way the program should 
behave. Contrasts between observed and expected behavior stimulate highly con- 

* Knowledge acquisition, in our paradigm, refers to thcf transfer of expertise from a human expert; 
to a machine. The machine acquires the human's knowledge in the form of concepts and heuristics. 
When the machine extends its initial knowledge by various learning methods, we refer to this as 
knowledge refinement Different researchers might apply the term knowledge acquisition to varying 
aspects of these ^processes. ; ' 

* We do not know if machine lisaming techniques will ever achieve sufficient levels of success to 
obviate the role of humans in such efforts. Thus, we see machines mora as calculating aids to humans 
'than as standalone investigators of complex domains. For the foreseet ble future, at least, humans will 
play a major role in guiding deductive processes that the machines execute more rapidly and systemati- 
cally than would otherwise be possible. . 
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strained searches for underlying deficiencies in the knowledge base. These deficien- 
cies, in turn, suggest knowledge refinements. The refinement processes of diagnosis 
and knowledge modification are described in Sec. III. 



CONVERTING KNOWLEDGE INTO AN EXECUTABLE PROGRAM 

We believe that a very large clads of intelligent systems can be specified quite 
~ea^silyrA~beh"avioral"d^^ actions 
as well as prescriptive methods for attacking problems in the domain. Our ap- 
proach rests on the central idea of expressing these behavioral constraints and 
heuristics in terms of natural domain concepts, A description of mathematical 
discovery methods, for instance, should employ terms familiar to a mathematician. 
When we ask a matheTiiaticiaii for advice aboi his problem, such as what to do 
or when and how to do i^, wo should allow him to talk in his own terms. Asking 
him to express his knowledge in terms familiar to computer programmers forces 
him to translate mathematics into programming. On the other hand, asking a 
programmer to bridge the gap between the mathematician and the computer re- 
quires him to translate between cultures— he must first comprehend much of the 
field of mathematics and then map its concepts and methods into his own repertoire 
of computer capabilities. 

To obviate the need for cross-cultural translation, , we are developing systems 
that will accept an expert's advice expressed in familiar domain concepts. To under- 
stand the advice, we need to know the meaning of each constituent concept, and 
we must transform higher-level advice into actual procedures that the computer 
can perform. Our approach aims to assimilate individual concepts as terms with 
formally defined properties. Then, we attempt to understand advice (e.g., con- 
straints among elements, or heuristic rules for goal-directed actions) as specific 
compositions of the constituent terms. At present, much of this overall process is 
understood, and some of it has actually been accomplished by our computer pro- 
grams. We will sketch the primary features oflthis approach in the following 
paragraphs. (Detailed technical descriptions are given in Hayes-Roth et al., 1978; 
Mostow & Hayes-Roth, 1979a; Mostow & Hayes-Roth, 1979b.) 

From the general perspective, ' /2nou;Zedge programming converts adcice ex- 
pressed in soine naturalistic syntax into actions in the task environment. This 
requires several processes: /parsing the advice into syntactic structures; interpret- 
ing these structures by converting them to meaningful semantic representations; 
operationalizing the meaning structures by converting tliem into effective, execu- 
table expressions; integrating multiple pieces of advice into a coherent set of proce- 
dures; and, finally^ applying these procedures in actual situations to generate ac- 
tions.^ These operations use a knowledge base that stores individual domain con- 
cepts and their interrelationships (see Fig. 3). Assimilating each new piece of advice 

^ While all five ofthese subtasks are difHcult, we have ibcused our efforts on th§ problem of opetation* 
alization. Parsing and interpretation are fairly well understood as a result of the attention they have 
received in naturaManguage-understanding projects. Operationalization is a new topic of study. In the 
new paradigm, emphasizing rapid implementation of expert knowledge and rapid reimplementation of 
modified knowledge, operationalization plays a crucial role. Integration is a^so a very important and 
difficult problem. Unfortunately, we have not addressed this problem in a substantial way during the 
course of this research project. A neat solution to the integration problem would yield a single, compre- 
hensive program that could be applied simply, as if it were a typical computer procedure.! 
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requires the knowledge programmerto appreciate how the advice relates to other 
elements of the knowledge base. Ttus^the knowledge base bbth feeds the assimila- 
tion processes and represents the incremental additions they produce. * 

Several different tasks we have explored in this paradigm hiave exhibited 
remarkably similar properties. Each task emplosfs a small set of domain-specific 
concepts, a few constraints or rules, and ah open-ended set of prescriptiye heuristic 
m^ethods. Such problem domains include music composition, legal reasoning, tacti- 
cal planning, and game playing. We have found that a familiar card game, hearts, 
provides a good basis for illustrating the major points of this research (see also 
Balzer, 1966). The problem represented by the game of hearts is discussed below. 



EXAMPLES OF OPERATIONALIZATION 



What does it take to specify a program that will simply play hearts in accor- 
dance with the rules?* The published rules of the game define mandatory behavior- 
al constraints. A program that behaves in accordance with these rules will play a 
legal, albeit poor, game. A few simple rules will illustrate the nature of this advice- 
taking problem. 

Players rule. The game is played by four players.*^ 

• . •■ ' / ■ ' ■ 

Players sequence rule. During a trick, players play . 

in clockwise order around the table. 

■ /, • / 
Trick rule. A trick is a sequence in which each player 

plays one card. 

Trick leader rule; The first person to play in the first 
trick is the one who has the two of clubs. The first \ 
/ player plays the twp of clubs. In other tricks, the \ 

winner of the preceding trick plays first. ^ 
j ^ i ; • ; . - ' . 

j- Follow suit rule. Each player, if possible, rhust play a 

card in the suit of the first card played in the trick, 

' , r ^ ' ' • ' 

I Win irick rule. In each trick, thie player who plays the / . 
r ' hijghest card in the suit lecl'wins the trick. \ 

The processes of parsing and interpreting these rules would generate knowledge- 
base! elements of the following sorts:® 

i Players rule. • : 

I Players. = |pl, p2, p3, p4[ : 

I Players Sequence rule. 

I if pi has jusf played in a trick and p2 has not played 
I I in the same trick then p2 plays next; . 

if p2 has jiist played.jn a trick and p3 has not played 
' in the same triek then p^ plays nfext; 

I if p3 has just played iri a trick and p4 has not played ' 
I in the same trick then p4 plays next; 

\ if p4 has just played in a trick and pi has not played 
-in the.same trick then pi plays next; 

I . Tricfe rule. • / . 

if t is a trick 
then ql plays cl during t 

* The reader who is concerned with the qoncept rather than the, technical details of this problem can 
skim the technical material that follows without seriously affecting the continuity of the discussion. 

* For simplicity, we ignore variations oh this rule (for example, one variation allows a three-player 
game). ' ' ' 

* To avoid unnecessary formality and technicality, we have expressed propositions and conceptual 
statements in terms of simple English equivalents.>.Reader8 interested in our LISP-based representa- 
tions may refer to the technical reports cjted previously. 
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followed by q2 plays c2 during t 

followed by q3 plays c3 during t 

followed by q4 plays c4 during t v 

and |ql,q2,q3,q4[ = players 

and cl, c2, c3, g4 are elements of cards. 

Trick leader ru/e 
if and only if player p has card two of clubs 
then player p plays first in the first trick 

and player p plays the card two of clubs in the first trick; 

if and only if player p wins a trick t 

then player p plays first 'in the trick t' following t; 

Follow suit rule. 

if the first card c played in a trick t is of suit s 

and player p before playing in trick t has some cards in suit s 
then player p plays some card d in suit s during trick t; 

Win trick rule. 
if and only if 

the card c is the first card played in trick t . » 
and the suit of card, c is s 
and the cards played in trick t are called C : 
and C is the subset of C whose suit is s 
and the highest valued element of C is. card d 
and card d was played by player p 
then player p \yins trick t; ' 

The concept definitions supporting such interpretations are of two sorts. The 
first are domain-dependent definitions, such as card, game, hand, and play. For 
example, play would be defined aff* an action by a playfer that changes the location 
of a card from the hand of the player to^thie pot. The .posisible locations of cards 
include a pliayer's hand, a player*s pile, and^the pot. In pur work^" we use a iform of 
the lambda calculus (Church, 1941; Allien, 1978) for encoding these diefinitions. The 
second type of definition , is used for representing d6main*independent concepts, 
such ^ set, suhseti Sorney ex^ 

Let us suppose fiDi; the pungent example. that onlylthe preceding rules were 
specified, along with a minimal additipnal set prescribing how players receive jbheir 
initial hands and.how the winner.of each trick takes the cards in the pot. Gould we 
directly apply the knowledge to produce behavior? The answer ia no, because th 
constraints recognize acceptable behavior without telling us how. to g^snlsriite it. 
That is, the constraints partially define what to do without explaining hbu;.'I)iScov- 
ering how to achieve such desired behavior requires uff to operatiohalize the advice. 
In effect, we need to convert the advice concerning what into executable mefhpds; 

Evenan this simple case, the search for effective methods to achieve the desired 
goal requires planning and problem-solving, which incidentally produce additional 
insights into the domain. 

The general methods we have employed for operationalization may be summa- 
rized as goal-directed, planning. We begin with a statement of desired behavior 
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which, in this r::ample, might be "player p plays a card c in the current trick t/* 
The problem hr. to convert this goal expression into a procedure that achieves the 
goal and satist .5 the constraints. Because this statemr^nt does not tell us how we 
may choose a sadsfactorycard, we regard it as an ineffective expression. We then 
attempt to transform this eixpfession into a composite of individual subexpressions 
where each component represents an effective expression; that is, each corresponds 
either to a known valuie or an executable procedure that can produce a value. 
Several different problems need to be solved, and we use a variety of different 
methods. 

In this simple example, we might attempt to transform the expression by 
adapting a general-purpose AI problem^solving method to this task. For example, 
we might attempt to adapt the general method of generate-and-test (Newell, 1969; 
Newell & Simon, 1972). To exploit this method, we need only find some generator 
that suggests each possible action and then apply some effective procedure for 
verifying that the suggested action satisfies the constraints. This approach would 
lead us to an operationalization such as the following: 

Plan 1: 

consider each card c in turn; 
as?»'"a^ you were to play c; ^ 

if " ^. cai? prove that c would satisfy the constraints, 
tjsn piay c. • 

This plan ha^ rwo interesting aspects that reflect some deep and recurring issues. 
First, formulating a plan of action without having the actual situation data in hand 
requires very general and abstract reasoning. When we do not know exactly which 
cards have been played and which cards are in a^player's hand, for example, we 
have difficulty formulating any specific action. In an actual situation,- however, we 
might find a similar judgment straightforward. For example, we could easily prove 
that playing the three of clubs is legal when the two of clubs has just been played. 
These two alternative tasks— forming general plans before situations unfold, vs. 
planning on the spot— We distinguish as ordinary operationalization vs. dynamic 
operationalization. The first point then is that in most circumstances, dynamic 
operationalization seems both easier and more effective. Its primary disadvantage- 
arises from the need to postpone planning until actions are required. This means 
that, at each point, the system cannot act until it has thought through the issues. 

The second point concerns the degree of effectiveness we deman:! il*om an oper- 
ationalization. The general plan proposed above may or may not be effective. It 
presupposes two capabilities: (1) generating each possible card in tuiu apd (2) 
proving that the card satisfies the constraints. While in simple cases like the hearts 
task we can assuredly achieve the first capability, we can rarely establish valid 
proofs of such abstract propositions as in the second. The difficulty arises from the 
complete generality of the assertion to be proved, i.e., "c would satisfy the con- 
straints." To certify Plan 1 as effective, need a procedure that we know can prove 
such a-^sertions. Thus, we might simply assume the use of a general theorem- 
proving procedure to perform this subtask. This procedure, in turn, may or may 
not be assuredly effective. The overall degree of effectiveness of Plan 1 would then 
depend on the theorem prover's own effectiveness and the eventual success or 
failure of its theorem-proving efforts. 
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The point we wish to make in this context concerns the degree of effectiveness 
we ascribe to an operationalization. Two kinds of uncertainties ordinarily preclude 
developing assuredly effective plans. First, most real-world tasks address inherent- 
ly uncertain environments. In these tasks, the constraints and behavioral heuristics 
may not be atrictly provab! j. Worse yet, in tasks that require both reward-seeking 
and risk-avoidance behaviors (e.g., playing hearts), knowledge usually prescribes 
simultaneously opposing, hence inconsistent, objectives. Second, plans themselves 
introduce a different type of uncertainty. This uncertainty, considered above, arises 
from the residual ineffectiveness or incompleteness of the operationalization. We 
have found that both kinds of tmcertainties require heuristic solutions. That fs, 
incelligfent systems can reason informally to partially control uncertainty but they 
cannot eliminate it completely. Much of our research revolves around the nature 
of the heuristics both human experts and knowledge programmers can employ to 
control such uncertainty efficiently. 

Because Plan 1 represents quite a "weak" or unspecific operationalization, we 
might reasonably seek a stronger, or more detailed solution. A more interesting 
plan for playing a card can be derived by adopting another general AI problem- 
solving approach. In this case, we view the proble.m from the perspective of the 
general heuristic search method. This method specifies that to reach a goal state 
G from an initial situation S, we should choose actions from some set A that 
successively change attributes of S until they satisfy G. To apply this general 
method to the problem at hand inquires matching aspects of the given problem to 
components of, the general approach. What elements of the current problem corre- 
. spond to G, S, and A? The initial statement corresponds to G: Reach a^state where 
player p plays a card c that satisfies all of the constraints. Situation S corresponds 
to a set of quite general assertions about what is known^ such as that player p has 
cards C. The set ^of possible operations A is not immediately apparent. 

To transform an initial state S into a goal state G. we employ three different 
kinds of transformations: logicaVheuristic, and instrumental. Logical transforma- 
tions convert an initial expression into a logically equivalent one. One colrjmon kind 
of logical transformation is a s3mibolic type of case analysis. 'To analyze an ex- 
pression, we reexpress it as a set of alternatives, each of which rests on an addition- 
al, distinctive assuniption characterizing that particular case. For example, we 
might reexpress the assertion that p plays card c as a disjunction of two cases: (1) 
p plays first in the trick and plays card c; or (2) some other player p' plays first in 
the trick, and later in the trick p plays card c. This transformation preserves the 
truth value of the initial expression, but, more importantly, it suggests lone promis- 
ing way to divide and conquer the initial problem. 

' Case analyses break a single general problem into separable subprobjems and 
at the same time further characterize aspects of the task situation that bear 
dirfectly upon constraint satisfaction. In JCase-2,- where p -is assumed to playafler.. 
the tri^l«!^as begun, p must choose a card in the same suit as the first one played. 
In this exkfiipie, the Case 2 assumption can key the knowledge programmer to 
apply the foltaw suit rule in subsequent operationalization of this path. 

Heuristic trahs|brmations make plausible, if not necessarily valid, substitu- 
tions in expression^. Fqr example, in developing a plan of play with attention to 
likely effects, we might transform the initial expression **afler the trick is opened, 
p plays card c" into the twofpllowing cases: (3) ''after the trick is opened, p plays. 
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card c, and c is the trick-winning '^ard," or (4) "after the trick is opened, p plays a 
card whose suit is different from the suit of the first card played." While.these cases 
are verifiably exclusive,^ they do not exhaust the possible situations that might 
arise. Hence, a traneformation of this sort might be plausible but not equivalence- 
preserving. We have found many practical situations where such heuristic transfor- 
mations can lead to clever, if somewhat incomplete, plans. In this illustration, for 
example. Case 3 leads to application of the win trick rule, further specifying the suit 
and value of card c. Case 4, on the other hand, suggests an opera tionalization that 
confronts the {ollow suit rule. The prerequisites for that rule in turn become addi- 
tional conditions on this line of reasoning. 

The third type of transformation to expressions models the result of instrumen- 
tal action in the task environment. In the initial problem of generating a legal play, 
the only actions known to the system would be dealing, playing, leading, and 
winning a trick. Each of these corresponds to a transformation that .will affect the 
description of any supposed game situation. In the analysis of Case 3 above, in 
addition to inferring requisite properties of the winning card c, we can also deduce 
that p moves cards from the pot to his pile. In this simple example, the only 
instrumental action that sheds much light on operaiionalizing the initial expression 
is the primitive action of playing a card, i.e., moving a card from a player's hand 
to the pot. Later, however, the actions of winning a trick and moving cards from 
the pot to players' piles will play significant roles in operationalizing behavioral 
heuristics suggested by experts. 

The knowledge programmer represents a variety of general reasoning methods 
as trarrsformations and applies these to convert high-level objectives into corre- 
sponding effective procedures. The reasoning methods exploited include case analy- 
sis, partial-matching between an expression and the description of a general meth* 
od to guide attempts at adapting the general method to the specific problem, sim- 
plificatiop of complex expressions, approximation of uncertain or combinatorial 
alternatives, and, reformulation of an expression in terms of other known concepts. 
Each specific operationalization task requires some or all of these methods. For 
example, if the-knowledge progjanimer adapted the heiiristic s^ the 
task of playing aiegal card, one resulting operational expression foir"p plays card 
c and satisfies the constraints" would correspond to the following: ' 

Plan : . , / \ ^ 

if p plays first in the trick \ 
and this is the first trick 
and p has the two of clubs 
^ then p plays the t^yo of clubs. 



else if p plays first in the trick 



and p has a card c whose suit is s 
then p plays c, . 



else if card c' wa's the first card played in the trick 



lUiU the '^uit of c' is s 



and p has a card c whose suit is s 
then p plays c, ^ 




else if p has a card c 
then p plays c. 

Of course, because we have thus far considered only the mandatory constraints on 
behavior, no expertise has been included in this initial set of advice. In addition to 
necessary conditions on behavior, the kinds of heuristics we want to acquire 
directly from experts tell the program how it shoiMhehawe. This goes beyond the 
notion of behaviorahacceptobi/ity to the co^cept of desirability. As anyone familiar 
with law, music, hearts, and most other difficult tasks realizes, the bulk' of human 
knowledge in these domains directly concerns such prescriptive heuristics. 



OPERATIONALIZING STRATEGIl^TACTICS, AND PROCEDURES 

In a game like hearts, where the real, objective is to minimize winning tricks 
that contain point cards, expert advice concerns strategies, tactics, and procedures 
that can help reach this objective. The rules of the game reward some kinds of risk 
aversion and sonie kinds of risk-seeking behaviors. For example, a player can 
improve his.or her (relative) score either by taking fewer points than t^rie opponents 
or by taking all the points in a roixnd. Thus, a very simple' type of heuristic advice 
might be to "avoid taking points." We will consider ^lis example briefly to convey 
the nature of the knowledge-programming problem it exemplifies, (i^ detailed teich-' 
nical discussion of this particular example appears in Mostow' & Hayes-Roth,. 
1979b.) ■ _ ■ _ "■ \:-- : ■ : 

Before proceeding with the exaiiiple, however, we need to postulate a few more 
bits of knowledge. We will assume the knowledge prograniiner has assimilated the 
following facts: Any card that is in the suit of hearts has a point value of 1, and the 
queen of spades has a point value of 13. We assume also that the concept "take" 
has been defined to mean that a player winning a trick t takes aU cards playedin 
that trick, i.e., he moyes them from the pot (cards played in the trick) to his pile. 
The concept "avoid a n event x "'is defined to mean "prevent event x" or "iachieve 
not [x]?' Using^such'basic^^ 

ineffective advice "avoid taking points". into ah eflTective-procedure; It generates a 
plan that recreates the typical high-level steps most peo^jle apparently follow, 
although it works through many . more and lower-level steps than people conscious- 
ly make. ^' ;. 

Given the rules of the game and the advice to "avoid taking points," people 
reason roughly as follows: (1) Taking points means taking cards with pqint values; 
(2) the only way to take cards is to win a trick; (3) "avoid talking points" thus means 
not winning a trick; (4) this in turn, means playing a card that is not the highest one; 
and (5) this {Suggests playing the lowest card in your hand.^ 

, For the sake of brevity, we summarize, the actual machine-aided dexlvation at 
the same high level as the introspective human analysis. First, the program logical- 
ly transforms "avoid taking points" by substituting for the ferni "avoid" its literal 
definition. This produces an expression like "establish not [player p takes points]." 
Although the objects of "take" adtions are cards and cards can have points, it is 

. ■ . \ ^ - ' ' ■■ 

After a little thought, people often notice ways to improve this plan, but we shall reconsider those 
kindsof insights later when we discuss the.role of plan evaluation as a source of knowledge refinement. 
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impossible to "take" points directly. The program reasons heuristically that "tak- 
ing points" seems equivalent to "taking cards that have points." From this, the 
program notices tha^^^a sufficient condition for taking points is winning a trick in 
which some of the cards have points. To preclude this from happening, it reasons 
that negating any of the necessary conditions should do. It then produces a new 
expression that corresponds to "do not win a trick." It uses the constraints on trick 
winniny to infer that the player wins only if he plays the highest card in the same 
suit as the card first played. Finally, it reasons instrumentally that this condition 
would not occcr if he played a card of lowey value. This type of plan leads directly 

to a corresponding procedure for applying the advice. ~ " — — 

In a similar way, the program has been used with human assistance to produce 
plans for other kinds of advice in this game^. For example, one useful heuristic for 
new players is to "flush the queen of spades," i.e., force another player to play it. 
The kind of reasoning the program uses to develop its plan is as follows: By sub- 
stituting the definition of "flush," it infers that it needs to establish the condition 
"some player p must play the queen of spades." It uses its concept of "must" and 
the follow suit rule to infer that this objective requires that p have only one legal 
card to play — the queen of spades. This in turn entails either (1) p has only the 
queen of spades or (2) player q leads a spade, and p's only spade is the queen. It 
focuses on the second case and then develops a plan for how player q could force ' 
such a situation. In brief, it develops a plan for q to win a trick to take the lead. 
Then as long as q retains the lead, q continues to lead spades. As players familiar 
with the game will realize, this is an eflfective method for flushing the queen.® 



INTEGRATION 

We have done little thus far to address the question of integrating a variety of 
separate pieces of advice! This type of problem lends itself to two approaches. The 
first aim^ at an overall consistent integration^ while the second presumes no such 
comprehensive integration is feasible. As in tHfe-pgeceding examples, operationaliz- 

ing a single rpiece qf- advice-dften requires simultaneously satisfying numerous 

constraints. Such an approach to- comprehensive integration fits the overall frame- 
work we have illustrated throughout this section. In tqe second type of approach, 
we presumably do not know the ways in which several pieciss of advice ihteraQt or, 
worse yet, the ways in which independent pieces may contradict one another. This 
type of situation arises when we adviso, for example, both "avoid taking points" 
and "take all the points" or "take at least one point if no one else has." 

Our approach to integrating multiple pieces of advice takes two basic forms. 
First, we try to.formulate independent recommendations that themselves may 
become the objects of metaheuristics. That is, we wish to accept advice about when, 
how, and why to combine or favor one heuristic over another. Second, we want to 
infer these dependencies by understanding why ^ome heuristics produce undesir- 
able results in actual situations. In such cases, we wish to eliminate the anSbiguity . 
by refining the initial heuristic to restrict its application to appropriate situations. 
This kind of refinement is discussed in Sec. III. 

■ Readers may also develop variations of this plan that seem superior. Such variations are discussed 
in more detail in Sec. III. 
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In .sum, we have presented our paradigm for knowledge -programming and 
iterative reprogramming of intelligent systems. This section has focused on knowl- 
edge-progi'amming processes, which are pez'tinent to both initial prpgramming and 
recurrent reprogramming. The next section motivates and explains the reprogram- 
ming problem in mort detail. We have briefly explained the kinds of advice we 
expect our systems to assimilate and a variety of methodg for converting the advice 
into operational programs. In this process, we see that the knowledge programmer 
formulates plans that develop its initially vague concepts into effective procedures 
for accomplishing goals. These plans also play a major role in identifying weak- 
nesses in knowledge that stimulate learning. 
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lit PLAN EVALUATION AND KNOWLEDGE 
^ \ IIEFINEMENT 



/ 



BUGS REVEALED IN PROGRAM EXEGUTION/ 



To convert constraints and heuristics into actio^, the knowledge programmer 
develops a plan that integrates task-environment actions along with logical or 
heuristic inferences. In planning, the knowledge/pi'ogrammer reasons about the ^ 
effects of the ^arious trariisformations it emplpys. Some of its transformations 
preserve logical equivalence, while others introduce approximate or plausible rea- 
soning. Tliese latter kjjnds of transformations may introduce undesirable 'effect? or 
"bugs." The second phase of intelligent system 4evelopment is concerned with the 
identificaiion, diagn^is, and elimination oC si^fch .bugs. ' 

We have developed a list of bugs that anse in knowledge programming. Some- 
of these arise frpm omissions, errors, or ambjigfuity in the initial knowledge, while 
others are introduced by the knowledge-programming process. Table 3 summarizes 
these bugs. (Other AI reseai'chers have considered bugs in problem-solving proce- 
dures, but these hiarVe little in pmmon m those under consideration here; see, 
for example, Davis, ,1978; Davis, i979;yM;iler .& Gioldstein, 1976; Sussman, 1975; 
Sussmann & StallmanV 1975; Brown &^BM 




liable 3: • . % < 
Bugs Arising frq^ Knowledge Programming 



Type of Probiein 



Soutee of rrobiem 



Manifestation 



1. Excess generality 

2. Excess specificity • 

3. Concept poverty 

4. Invalid knowledge 

5. Ambiguous knowledge 

6. Invalid reai^oning 

7. Inadequate integration 

8. Limited horizon 

9. Egocentricity 



Specif cases overlooked' 

* Generality undetected. 

Useful relationship not 
dete'cted'^nd exploited. 

Misstatement of fact? or 
approximations. 

jmplicit 'dependencies not 
' adequately articulated. 

^Programmer incorrectly 
I transforms knowledge. 

/ Dependencies among 
I multiple pieces of advice 
I incompletely integrated. 

■J Consequences of recent 
I / past or probable future 
; events not exploited.. 

; Little ati^Rtibn paid. to 
probable meaning of ' 
others' actions. 



Good rule occasionally pro- 
duces bad effects. 

i Rules fail to cbVer^ enough 
>pases. 

Limited power and capability 
of system. 

Expert's expectations violated 



Cohflicts arise in some situa- 
^tiohs about what is best to do. 

Knowledge programmer's ex- 
pectations violated. • 

Rejec};ed - action alternatives, 
actually, satisfy more criteria', 
than selected action does. - 

Judgmental logic seems static^ 
not sensitive to changing or 
foreseeable situations. ' 

No apparent adaptation of 
onfe's.' behavior to exploit 
knowledge of other's '^Is^ns. 
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The nine problems listed in Table 3 span a large set of potential weaknesses in 
intelligent programs. For the Sake of. brevity, we will consider only one of these 
bugs in detail as an illustration of knowledge-refinen>ent techniques. We have 
chosen invalid reasoning^ because it illustrates many of the ideas that recur in 
knowledge refinement. After we have explained our approach to debugging reason- 
ing problems, we will briefly characterize the approaches taken for the other kinds 
of problems as well. 



DIAGNOSING AND FIXING A REASONING ERROR 

We address here only a limited class of reasoning errors, namely those which 
manifest themselves as discrepancies between the observed outcomes of e>^ecuting 
a plan and the expected outcomes. Here we are focusing on the expectations that 
the knowledge programmer generates as by-products of its opferationalizations and 
integrations. The knowledge programmer acts as if it believes the transformations 
used to convert. ineffective statements to specific procedures will produce results 
satisfying the original objectives. This belief applies, in turn, to each succ^^^ive 
transformation applied during the planning process. However, the transformations 
niay in fact yield .procedures that do not always satisfy these expectations. 

The approach we take to knowledge refinement in this type of problem begins 
with an attempt to analyze aii unexpected event. Thus expectations motivate and 
trigger the knowledge-refinement ipf^cess, as shown in Table 4. By analyzing the 
violated ej^pectation, we identify botJi what went wrong and why. Then, we propose 
changes to the underlying knowledge to remedy the problem. The success of this 
method often depends on isolating missing, extraneous, or imprecise predicates 
used to restrictsthe time at whicbsome action occurs. (This approach parallels that 



Table 4 

Knowledge-Refinement Approach 





Step 


' r ^^ ^^ Source of Mechanism 


1. 


Establish expectations 


During knowledge programming, planning establishes plausible 
antecedents and consequences of actions; these beliefs represent . 
expectations. 


' 2. 


Trigger* analysis 


When an actual Qventjviolates an expectation, the reasoning be- 
hind the expectation is reanalyzed i;i light of observable data. 


'3. 


Locate faulty rules 


A set of diagnostic rules debug ^e planning logic by contrasting 
the a priori beliefs'with actual data. If a heuristic rule used by 
the plan assumes a false. premise or entails a false conclusion, 
it is faulty. 


4. 


Modify faulty, rules 

' . ■ ^ 


A set of learning rules suggest plausible fixes to the erroneous 
heuristic rule. These might alter its preconditions, assumptions, 
or expectations to keep it from 'producing the same faulty re- 
sult in a subsequent situation. 


. 5. 


Reimplement and test 


Incorporate a iiiodified heuristic rule into a new system by re- 
invoking tiiie knowledge programmer. Verify that the . rule 
eliminates the previous problem and test it in new 'Situations. ^ 
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discussed in Lakatos, 1976.) This vdll become clearer in the context of a concrete 
illustration. 

As one example of the act-evaluate-refine process, consider what happens when 
the machine attempts to execute the previously developed plan to flush the queen 
of spades. The plan was, roughly, take the lead, then lead spades until a player is 
forced to play the queen. Suppose that this plan worked well in several games, but 
during one game a sequence like the following unexpectedly occurred: The machine 
player wins a trick. It then leads the jack of spades, and the other players follow 
suit. The queen Is still held by one of the players. On the next trick, the machine 
Chooses another spade to lead. This time, it has only two spades left, the four and 
the king, and it chooses arbitrarily to play the king. The next player plays the five, 
the one after him plays the qiieen, and the last plays the ten. The machine has just 
won a. trick according to its plan, and it has even flushed the queen. Unfortunately, 
it has also taken 13 points, presumably a very undesirable outcome. 

What might a person in the machine's situation do at this point? With apparent- 
ly little effort, a person would recognize that the plan was buggy, because it 
achieved an undesirable result that was unexpected. Implicit in the plan was the 
notion that the player with the queen would be coe^-ced into playing it and, presum- 
ably, winning the spade trick with it. In response to this insight, a hxunan player 
would amend the plan appropriately/ The fix in this case wouldirequire that when 
trying to flush the queen, a pliayer must leaid only spades below the queen. 

Our learning methods capture the general logic behind this type of analysis. 
There are many chains of reasoning that might lead to the same, proposed refine- 
ment as our hypothetical human produced. We will explain one type of argument 
that appears programmable. . ^ , - 

Let us suppose that the machine (unlike a human) has no precise expectation 
regarding the queen-flushing plan. However, since it followed supposedly expert 
advice, it has a generaVexpectation that bad consequences should not result. When, 
as in ;this case, undesirable results occur, the program' tries to understand why it 
suffered such an . outcome and how it could have prevented it. _ ^ 

The machine ariiatyzes the last trick to infer cause-effect relations, based on its 
current knowledge. To take 13 points in the trick, it had to v/in the trick during 
which the queen was played. So it conjectures for itself some refined advice: Flush 
the queen of spades.biit do not win a t ;ick in which the queen is played^, Because 
this refineu advice surpasses the original advice in quality, the machine has already 
improved its knowledge. On the other hand, this high-level advice requires ope^'^ 
ationalization if it is to be useful. However, our current knowledg^^ programmer 
does correctly opera tionalize this advic' by producing a plan'eorrespdnding tp the 
following: Take the lead, then continue leading spades below the queen. Thus, this 
type of bug is eliminated by formulating a desired refinement directly in terms of 
a new high-level prescriptive heuristic. The refined heuristic, in turn, is implement- 
ed by the same knowledge-programming methods previously used for accepting 
advice from humans. (In some cases, as in this example, the refined heuristic can 
also be implemented simply by modifying the previous plan, as opposed to starting 
over from scratch.) 

To continue our illustration, let us suppose that the machine begins to apply its 
refined plan. Because it knows that the' plan has been refined to prevent it from 
taking the queen of spades itself, it notes this specific expectation in the knowledge 



base as a predicted consequence of the plan. In a new game, however, suppose it 
has the ten, jack, and king of spades. It wins f. trick, then leads the ten. All players 
follow suit with lower cards, so the machine leads again with the jack. Again it wins 
tlie trick. At this point, its revised plan proscribes leading spades, so it plays a 
diamond. Another player wins the trick, and continues to lead spades. The machine 
is forced to play the king, and the player after it follows suit with the queen of 
spades. 

Again, contrary to its specific expectation, it wins the trick and takes 13 points. 
Now it attempts to discover why its expectation was violated. It constructs a cause- 
effect model of the events leading to the disaster. In this model, it notes that at the 
time it played the king, it had no other choices. So apparently, by that time, only 
by keeping the other player from leading spades could it have prevented the 
disaster. Alternatively, it reviews events prior to that trick to see what, if anything, 
it. did that contributed to creating a situation where no options existed. It notices 
that playing the ten and jack of spades earlier produced the state where it had only 
the king of spades. It notes that these actions were taken with the express intention 
of preventing it from taking the queen, but apparently they contributed directly to 
just that outcome. 

It now proposes to itself another refinement. It should prevent a reoccurrence 
of this type of situation in the future. Its proposed advice: Do not lead low spades 
if you can be forced to play a spade higher than the queen. ^ This, in turn, leads to 
an operationalization that requires an estimation of the probable distributions of 
spades among players. While we have developed some methods for handling such 
probability functions, we have not yet implemented those needed for this particular 
problem. However, as persons knowledgeable in the game will notice, the proposed 
concept of a card that is "skfe" vi§-a-vis the opposing distributions is quite sophis- 
ticated. In fact, generalizations of this "safe spade" concept, such as "safe in suit 
x" or "safe with respect to all suits," play major roles in expert strategies. " 

As another example of knowledge refinement, consider again the plan devel- 
oped in Sec. II to avoid taking points. That plan proposed playing the lowest 
possible card. Using this plan, the machine expects it will avoid taking points, but 
there are numerous ways that the plan leads to violated expectations, each of which 
reflects characteristics similar to those in thi^ queen-flushing examples. For exam- 
ple; it may play its lowest card (a five, say) and still win a trick with points. This 
causes it to weaken its expectations (i.e., to associate some uncertainty with this 
predicted outcome). Pursuant to.such a play, it may take another trick with its 
current lowest card (a ten, say), a^^n with points. However, if it had played the 
ten before the five, it might have avoided winning the second trick, because in the 
second trick the five might have befen lower than another player's card. Each of 
these problems gives rise to new attempts to refine both the expectations and the 
plan, in a manner similar to that previously described. 

* This example has pot actually been performed by a machine implementation. Before it coiild be 
implemented, several difficult issues would arise. Foremost among these, the diagnostic system wpuld 
need to conjecture several alternative problems and solutions. Each of these proposed solutions would 
require, in turn, experimental testing through additional play. For example, the program might have 
hypothesized the remedy. Do not begin to flush the queen of spades if you cannot retain the lead. This 
heuristic seems beneficial, but we cannot be certain. Empirical validation of dtemati ve heuristics seems 
unavoidable. 
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Our general knowledge-refinement strategy can be characterized simply, as 
shown in Fig, .4. ■ 

The contrast between expectations and actual outcomes focuses the learning 
system directly on specific problems. The system then attempts to find the flaws in 
its original causal model in light of the new data at hand. This in turn suggests 
additional conditions or new goals for knowledge programming. 

The overall approach we have taken to this problem employs three basic ele- 
ments: (1) proofs, (2) diagnostic rules, and (3) learning rules. While these steps have 
not actually been implemented on a computer, we have hand-simulated all of them. 
The knowledge proj^ammer associates with each plan and its expectations a proof 
(or an informal rationale). TTie proof of a plan links assumed conditions to expecta- 
tions by follo>ying paths representing the equivalence of logical transformations, 
the plausible sufficiency of heuristic transformations, or the antecedent-consequent 
relations of instrumental acts. At each point, a transformation links premises to 
expectations, and these expectations may become part of the premises for a later 
inference. In short, a proof maps a general model of cause-effect relations into a 
specific derivation of the expected consequences of the planned actions. , 
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Pig. 4— Knowledge-refinement strategy 
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Diagnostic rules examine the proof in light of the evidence and identify hypo- 
thetical deficiencies in the knowledge base. A typical diagnostic rule is as follows: 

Invalid Premise Diagnostic Rule. If an expectation is violated, find a premise in 
the proof of the expectation that is falsified by the data. If the false premise 
follows from some inference rule whose own antecedent premises (necessary 
conditions) are true, declare that rule faulty. 

Learning rules, on the other hand, specify ways to modify heuristics to correct 
deficiencies. We have generated a large set of such niles to date. Two examples of 
learning rules are given below: 

Require Implicitly Assumed Premise Explicitly. If an implicit assumption of a rule 
is falsified during proof analysis, add the premise to the required conditions 
of the rule and delete any other premises that it implies. 

Guarantee Assumed Conditions. If an assumed premise is falsified during proof 
analysis, identify sufficient conditions for its validity and make these required 
conditions for the associated plan component. 

Figure 5 demonstrates how these diagnostic and learning rules are used to refine 
the original ''flush the queen of spades" plan as discussed above. Figure 5 also 
exemplifies the knowledge-refinement approach outlined in Fig. 4. 

We have thus found many ways to evaluate a plan against observable outcomes 
to identify weaknesses, coryecture refinements, and evaluate these refinements 
experimentally. Very little of this work has been implemented, because of the vast 
number of possible learning strategies (see Table 5) and the wide variety of specific 
possible applications. Any eflTorts to implement these concepts in a realistically 
complex task will encounter considerable combinatorial difficulties. Each error may 
suggest several hypothetical bugs an'd fixes. Each of these will require independent 
empirical (or formal) validation, usually accomplished best by experimental testing. 
The need for testing hypothetical concepts and rules will lead to alternative knowl- 
edge bases and associated operational programs. Multiple systems of this sort are, 
of course, difficult to manage even in liniited foflware-development environments. 



SUMMARY 

Once a plan is executed, much can be learned from a retrospective analysis. 
When advice is provided initially, two important things are niissing that later 
support evaluation and discovery. The first new source of information is the actual 
situation description. The details of the actual situation in which the plan executes 
reveal and implicitly define important special cases that the general operationaliza- 
tion overlooks. Second, having acted, we can see the true effects of our behavior 
on the environment This provides sources of confirmation or disconfirmation of 
parts of our plans, which then stimulate focused efforts at diagnosis and knowledge 
refinement. These provide numerous opportunities for concept formulation, and 
each, in turn, initiates a new cycle of knowledge acquisition, knowledge program- 



.ming, and knowledge refinement. 
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PLAN: 

Flush queen of spades: 

If player P takes the lead 

and . P doesn 't have the queen of spades 

then P continues leading spades 

Expectation: 

P doesn't take the queen of spades. 

Proof of expectation: 

1. Player P takes the lead 

2. P doesn't have the queen of spades. 

3. P continues leading spades. 

4. If player P takes the lead and P 
doesn't have the queen of spades 
and P continues leading spades 

then opponent will play queen 
of spades. 

5. Opponent will play queen of spades. 

6. If an opponent plays queen of sp ues 

then the opponent wms the trick 
and opponent takes- the queen of 
spades. 

7. Opponent takes the queen of spades 

8. If opponent takes the queen of spades 

then player P doesn't take quieen 
of spades. 

9. Player P doesn't take queen of spades. 



Premise (condition of plan) 
Premise (condition of plan) 
Premise (action of plan) 



Heuristic rule 

Derived premise from 1, 2, 3, 4 



Heuristic rule 

Derived premise from 5, 6 



Heuristic rule 

Derived premise from 7, 8 



Behavior in actual play: P leads king of spades; 

Opponent plays queen of spades. 

Outcome: P wins the trick; P takes the queen of spades. 

Expectation of "Flush queen of spades" plan is violated. 

Apply diagnostic rules to identify problems: 

Us&ng "Invalid Premise" diagnostic rule, the derived premise 
in Statement 7 is falsiHed by the data. The inference rule 
used to derive thin false premise is the rule specified in 
Statement 6. Its premise is true, but its conclusion is false. 
Declare this rule faulty. 

Apply Learning Rules to modify plans and heuristics: 

Using "Guarantee Assumed Conditions" learning rule, the 
system looks for other rules in the knowledge base that 
identify conditions for inferring Statement 7. It may find, 
for example, the inference rule: 

If opponent player plays a high card C 

and player P plays below C 

then opponent wins trick and takes C 

In our current example, C is the queen of spades. 
This rxile now replaces the faulty, rule in Statement 6 with 
the new premise 

Player P plays below the queen of spades 

added as a premise to the plan and the proof. The resultant plan is 

If player P takes the lead 
and P doesn't have the queen of spader 
then P continues leading spades 
below the queen of spades 



Fig. 5— Knowledge-refinement example 
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Knov/ledge-Refinement Strategies 





Type of Problem 


Refinement Strategy 


1. 


Excess generality 


Specialize the rules, using case analysis, proof analysis, concept 
hierarchy specializati6ns. ' . 


2. 


Excess specificity 


Generalize the rules, using equivalence of qases, proof analysis, 
concept hierarchy generalizations. 


3. 


Concept poverty 


Create new concepts, by characterizing a particular problem, 
adding its definition, consequences, arid proposed solution to 
knowledge base (e.g.,."sacririce," "safe" distribution). 


4. 


Invalid knowledge 


Correct faulty advice, .using proof analysis, diagnosis, and re- 
finement. 


5. 


Ambiguous knowledge 


£iApiorc oii^riiaiiyc tiivcrprebabionB anu prune' uiose uiat pro* 
duce least desirable effects. 


6. Invalid reasoning 


Correct faulty operationalizations, using proof analysis, diag- 
nosis, and refinement 


7. 


Inadequate integration 


Develop comprehensive operationalizations that satisfy multiple 
pieces of advice simultaneously; sequentially order separable, 
criteria to satisfy most important considerations fhrst. 


8. Limited iiorizon 


Elaborate plans to incorporate contingencies and predict, 
monitor, and remember their outcomes; wherever possible, pre- 
fer dynamic operationalizations to static ones. 


9. Egocentricity 


During planning, consider what others are likely to do; use your 
own plans to model what you would do in their places; then ■ 
monitor their behavior to aissess its consistency with your model. , 



IV. CONCLUSIONS AND RECOMMENDATIONS 



FINDINGS 

In today's environment, major advances in AI arise primarily in conjunction 
with knowledge-engineering research. In this area, the power of intelligent systems 
derivejs primarily from the knowledge of human experts. The primary bottlenecks 
in the construction of intelligent systems are formulating knowledge for program- 
mers, converting the knowledge into effective procedures, and iteratively evaluat- . 
ing a program's behavior, modifying the knowledge, and reiniplementing the corre- 
sponding program code. 

We have formulated a framework for exploring solutions to these problems 
^which provides a basis for experts to express domain knowledge in terms of natural 
domain-specific concepts. This requires a formal knowledge^representation scheme 
and a substantial set of built-in primitive concepts from which the specific domain 
concepts are constructed. Once the concepts are defined^ the expert can express two 
kinds of advice about the behavior of the program. Constraints specify restrictions 
on allowable behavior, while heuristics prescribe desirable modes of behavior. 
These may be ambiguous, incomplete, or even inconsistent. 

This advice is converted into a working program through a process of oper- 
ationalization, which transforms constraints and heuristics into eflfective proce- 
dures. In this process, the current program uses the expert's supplied knowledge 
along with about 300 transformation rules. Some of these reformulate expressions 
in equivalent terms, for example, by substituting a definition for some specific term. 
Some of the rules prescribe sufficient or approximately sufficient means of achiev- 
ing ends. -Finally, the operationalization process uses instrumental reasoning to 
predict effects of potential actions or to reason backwards from desired effects to 
sufficient conditions and actions. 

In the process of operationalizing advice, a plan is develpped that prescribes a , 
sequence of actions required ^to accomplish the goals and satisfy the constraints. To 
develoj) this plan, the knowledge programmer employs a causal model to establish 
a "proof of the plan's expected eflfects. When the plan is executed, new data about 
the situation and the effects are obtained. By contrasting observations with expec- 
tations and premises in the proof, diagnosis rules indicate faulty components of the 
plan. These in turn lead to plausible refinements to the plan and corresponding 
changes to the knowledge base. These refinements, in turn, reinitiate the cycle of 
operationalization, execution, and evaluation. 

We have found this paradigm quite valuable as a source of new ideas and 
methods for knowledge acqiiisition and refinement. We have implemented only the 
operationalization component and have experimented with several knowledge rep- 
resentations in different tasks to develop diagnostic and leariiing rules. We have 
not yet converged on a small set of rules for any aspect of this paradigm. We have 
approximately 300 rules of operationalization for two tasks (hearts and a simple 
music composition task), and fewer than 100 diagnostic and learning rules. How- 
ever, we foresee these numbers increasing to as many as a few thousand.. For 
example, many of Lenat's proposed general concept-discovery methods (e.g., gener- 
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alization and specialization heuristics) seem to apply to behavioral tasks as well as 
to mathematics (Lenat, 1976; Lenat, 1977b). Our current ceilings have been im- 
posed by funding and personnel limitations: We have found many more interesting 
and productive lines of investigation than we have had resources to pursue. 



RECOMMENDATIONS 

We recommend that the proposed research paradigm be adopted widely.: It 
focuses on a set of learning problems that are considerably different from those 
addressed in most previous learning research in AI ind cognitive psychology. Much 
previous research (our own included) addressed isolated concept, pattern, and rule- 
learning problems — ^^tasks that seem fundamentally tied to limited applications. 
Although the number of potential applications for pattemt or rule-induction sys- 
tems is large, most learning probletns will arise in the context of more fully inte- 
- grated intelligent systems. In these systems, capabilities will be required for recog- 
nizing patterns, gathering information, assessing uncertainties, trading ofFbetween 
multiple goals, satisfying a variety of constraints, and dynamically applying gen- 
eral principles to specific situations! These capabilities in turn create demands for 
both rapid knowledge programming and rapid refinements? 

We also recommend that learning issues be approached within the broader 
context of purposive behavior. In this context, the value of knowledge derives from 
its capacity to. contribute to goal attainment. Goal-oriented planning provides a 
basis for contrasting the expected eflfects of knowledge with actual eflfects. This in 
turn dictates what new knowledge must be produced and how to integrate it into 
a preexisting knowledge base. This type of teleo logical orientation strongly moti- 
vates and guides knowledge acquisition and refinement.* 

Finally, we suggest an increased emphasis on the core research problems stand- 
ing between our current state of technology and the capability of automatic knowl- 
edge programming and refinement discussed in this report. The primary research 
problems include (1) representations for concepts, constraints, and heuristics amen- 
able to machine interpretation and semantic analysis; (2) translators for mapping 
natural domain descriptions into these knowledge representations; (3) operationali- 
zation and planning; (4) plan evaluation and proof analysis; and (5) knowledge- 
refinement and concept-discovery heuristics. 

* A corollary to this recommendation argues that when constructing or modifying AI programs, we 
sHould try to analyse the reasoning involved. One step in this direction is to identify operators for 
transforming specifications into working code. (See Balzer et a)., 1977; Barstow, 1977; Mostow & Hayes- 
Roth. 1979a.) 
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